Prompt Tuning of Deep Neural Networks for Speaker-adaptive Visual Speech Recognition

Visual Speech Recognition (VSR) aims to infer speech into text depending on lip movements alone. As it focuses on visual information to model the speech, its performance is inherently sensitive to personal lip appearances and movements, and this makes the VSR models show degraded performance when they are applied to unseen speakers. In this paper, to remedy the performance degradation of the VSR model on unseen speakers, we propose prompt tuning methods of Deep Neural Networks (DNNs) for speaker-adaptive VSR. Specifically, motivated by recent advances in Natural Language Processing (NLP), we finetune prompts on adaptation data of target speakers instead of modifying the pre-trained model parameters. Different from the previous prompt tuning methods mainly limited to Transformer variant architecture, we explore different types of prompts, the addition, the padding, and the concatenation form prompts that can be applied to the VSR model which is composed of CNN and Transformer in general. With the proposed prompt tuning, we show that the performance of the pre-trained VSR model on unseen speakers can be largely improved by using a small amount of adaptation data (e.g., less than 5 minutes), even if the pre-trained model is already developed with large speaker variations. Moreover, by analyzing the performance and parameters of different types of prompts, we investigate when the prompt tuning is preferred over the finetuning methods. The effectiveness of the proposed method is evaluated on both word- and sentence-level VSR databases, LRW-ID and GRID

Efficient and effective human action recognition in video through motion boundary description with a compact set of trajectories

Human action recognition (HAR) is at the core of human-computer interaction and video scene understanding. However, achieving effective HAR in an unconstrained environment is still a challenging task. To that end, trajectory-based video representations are currently widely used. Despite the promising levels of effectiveness achieved by these approaches, problems regarding computational complexity and the presence of redundant trajectories still need to be addressed in a satisfactory way. In this paper, we propose a method for trajectory rejection, reducing the number of redundant trajectories without degrading the effectiveness of HAR. Furthermore, to realize efficient optical flow estimation prior to trajectory extraction, we integrate a method for dynamic frame skipping. Experiments with four publicly available human action datasets show that the proposed approach outperforms state-of-the-art HAR approaches in terms of effectiveness, while simultaneously mitigating the computational complexity

Bioprinting of three-dimensional dentin-pulp complex with local differentiation of human dental pulp stem cells

Numerous approaches have been introduced to regenerate artificial dental tissues. However, conventional approaches are limited when producing a construct with three-dimensional patient-specific shapes and compositions of heterogeneous dental tissue. In this research, bioprinting technology was applied to produce a three-dimensional dentin-pulp complex with patient-specific shapes by inducing localized differentiation of human dental pulp stem cells within a single structure. A fibrin-based bio-ink was designed for bioprinting with the human dental pulp stem cells. The effects of fibrinogen concentration within the bio-ink were investigated in terms of printability, human dental pulp stem cell compatibility, and differentiation. The results show that micro-patterns with human dental pulp stem cells could be achieved with more than 88% viability. Its odontogenic differentiation was also regulated according to the fibrinogen concentration. Based on these results, a dentin-pulp complex having patient-specific shape was produced by co-printing the human dental pulp stem cell-laden bio-inks with polycaprolactone, which is a bio-thermoplastic used for producing the overall shape. After culturing with differentiation medium for 15 days, localized differentiation of human dental pulp stem cells in the outer region of the three-dimensional cellular construct was successfully achieved with localized mineralization. This result demonstrates the possibility to produce patient-specific composite tissues for tooth tissue engineering using three-dimensional bioprinting technology

Automatic 3D Model Generation based on a Matching of Adaptive Control Points

Abstract The use of a 3D model helps to diagnosis and accurately locate a disease where it is neither available, nor can be exactly measured in a 2D image. Therefore, highly accurate software for a 3D model of vessel is required for an accurate diagnosis of patients. We have generated standard vessel because the shape of the arterial is different for each individual vessel, where the standard vessel can be adjusted to suit individual vessel. In this paper, we propose a new approach for an automatic 3D model generation based on a matching of adaptive control points. The proposed method is carried out in three steps. First, standard and individual vessels are acquired. The standard vessel is acquired by a 3D model projection, while the individual vessel of the first segmented vessel bifurcation is obtained. Second is matching the corresponding control points between the standard and individual vessels, where a set of control and corner points are automatically extracted using the Harris corner detector. If control points exist between corner points in an individual vessel, it is adaptively interpolated in the corresponding standard vessel which is proportional to the distance ratio. And then, the control points of corresponding individual vessel match with those control points of standard vessel. Finally, we apply warping on the standard vessel to suit the individual vessel using the TPS (Thin Plate Spline) interpolation function. For experiments, we used angiograms of various patients from a coronary angiography in Sanggye Paik Hospital

Localization Uncertainty Estimation for Anchor-Free Object Detection

Since many safety-critical systems, such as surgical robots and autonomous driving cars, are in unstable environments with sensor noise and incomplete data, it is desirable for object detectors to take into account the confidence of localization prediction. There are three limitations of the prior uncertainty estimation methods for anchor-based object detection. 1) They model the uncertainty based on object properties having different characteristics, such as location (center point) and scale (width, height). 2) they model a box offset and ground-truth as Gaussian distribution and Dirac delta distribution, which leads to the model misspecification problem. Because the Dirac delta distribution is not exactly represented as Gaussian, i.e., for any $\mu$ and $\Sigma$. 3) Since anchor-based methods are sensitive to hyper-parameters of anchor, the localization uncertainty modeling is also sensitive to these parameters. Therefore, we propose a new localization uncertainty estimation method called Gaussian-FCOS for anchor-free object detection. Our method captures the uncertainty based on four directions of box offsets~(left, right, top, bottom) that have similar properties, which enables to capture which direction is uncertain and provide a quantitative value in range~[0, 1]. To this end, we design a new uncertainty loss, negative power log-likelihood loss, to measure uncertainty by weighting IoU to the likelihood loss, which alleviates the model misspecification problem. Experiments on COCO datasets demonstrate that our Gaussian-FCOS reduces false positives and finds more missing-objects by mitigating over-confidence scores with the estimated uncertainty. We hope Gaussian-FCOS serves as a crucial component for the reliability-required task

Hydrogen-bonded multilayer of pH-responsive polymeric micelles with tannic acid for surface drug delivery

We report the design of a platform for the delivery of hydrophobic drugs conjugated to block copolymer micelles via pH-responsive linkage that are assembled within hydrogen-bonded polymer multilayer thin films.close465

Formation of ZnO Micro-Flowers Prepared via Solution Process and their Antibacterial Activity

This paper presents the fabrication and characterization of zinc oxide micro-flowers and their antibacterial activity. The micro-flowers of zinc oxide composed of hexagonal nanorods have been prepared via solution process using precursor zinc acetate di-hydrate and sodium hydroxide in 3 h of refluxing time at ~90°C. The antibacterial activities of grown micro-flowers were investigated against four pathogenic bacteria namely S. aureus, E. coli, S. typhimurium and K. pneumoniae by taking five different concentrations (5–45 μg/ml) of ZnO micro-flowers (ZnO-MFs). Our investigation reveals that at lowest concentration of ZnO-MFs solution inhibiting the growth of microbial strain which was found to be 5 μg/ml for all the tested pathogens. Additionally, on the basis of morphological and chemical observations, a chemical reaction mechanism of ZnO-MFs composed of hexagonal nanorods was also proposed

Proteomic Validation of Multifunctional Molecules in Mesenchymal Stem Cells Derived from Human Bone Marrow, Umbilical Cord Blood and Peripheral Blood

Mesenchymal stem cells (MSCs) are one of the most attractive therapeutic resources in clinical application owing to their multipotent capability, which means that cells can differentiate into various mesenchymal tissues such as bone, cartilage, fat, tendon, muscle and marrow stroma. Depending on the cellular source, MSCs exhibit different application potentials according to their different in vivo functions, despite similar phenotypic and cytological characteristics. To understand the different molecular conditions that govern the different application or differentiation potential of each MSC according to cellular source, we generated a proteome reference map of MSCs obtained from bone marrow (BM), umbilical cord blood (CB) and peripheral blood (PB). We identified approximately 30 differentially regulated (or expressed) proteins. Most up-regulated proteins show a cytoskeletal and antioxidant or detoxification role according to their functional involvement. Additionally, these proteins are involved in the increase of cell viability, engraftment and migration in pathological conditions in vivo. In summary, we examined differentially expressed key regulatory factors of MSCs obtained from several cellular sources, demonstrated their differentially expressed proteome profiles and discussed their functional role in specific pathological conditions. With respect to the field of cell therapy, it may be particularly crucial to determine the most suitable cell sources according to target disease